Robot and housing

ABSTRACT

Convenience and usefulness of a tele-existence system are enhanced taking notice of the possibility by collaboration of tele-existence and a head-mounted display apparatus. A movable member is supported for pivotal motion on a housing ( 20 ). In the housing, a driving motor and a transmission member for transmitting rotation of the driving motor to the movable member are provided. A state information acquisition unit acquires facial expression information and/or emotion information of a user who wears a head-mounted display apparatus ( 100 ). A driving controlling unit controls rotation of the driving motor on the basis of the facial expression information and/or the emotion information.

TECHNICAL FIELD

The present invention relates to a robot and a housing of the robot.

BACKGROUND ART

A head-mounted display (HMD) is utilized in various fields. By providinga head tracking function to the HMD and updating a display screen imagein an interlocking relationship with the posture of the head of theuser, an immerse feeling in the video world can be increased.

CITATION LIST Patent Literature

[PTL 1] JP 2015-95045A

SUMMARY Technical Problem

In recent years, a technology called tele-existence that utilizes arobot disposed at a distant place as an avatar of a user itself hasappeared. If a robot at a distant place transmits image data or sounddata of its surroundings and the data are reproduced at on user side,then the user can communicate with people therearound with such arealistic feeling that the user is at the place of the robot.

The inventor of the present application took notice of the possibilityby collaboration of the tele-existence and the HMD and developed atechnology that improves the convenience and the usefulness of atele-existence system.

The present invention has been made in view of such a subject asdescribed above, and it is an object of the present invention to providea structure of a robot that is operated by remote control, a technologythat processes viewing data acquired by a robot, and a technology forusefully utilizing viewing data acquired by a robot.

Solution to Problem

In order to solve the subject described above, a robot of a certain modeof the present invention includes an actuator apparatus and a housingwhose posture can be changed by the actuator apparatus. The robot ofthis mode includes a movable member supported for motion on the housing,a motor provided in the housing, a transmission member configured totransmit rotation of the motor to the movable member, a drivingcontrolling unit configured to control rotation of the motor, and astate information acquisition unit configured to acquire facialexpression information and/or emotion information of a user who wears ahead-mounted display apparatus thereon. The driving controlling unitcontrols rotation of the motor on the basis of the facial expressioninformation and/or the emotion information.

Another mode of the present invention is a housing that includes amovable member supported for motion on the housing, a motor, and atransmission member configured to transmit rotation of the motor to themovable member. The transmission member is a member that deformstorsionally and elastically.

It is to be noted that also arbitrary combinations of the componentsdescribed above and conversions of the representation of the presentinvention between a method, an apparatus, a system, a computer program,a recording medium in which the computer program is recorded, a datastructure and so forth are effective as modes of the present invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view depicting an example of a configuration of aninformation processing system according to an embodiment.

FIG. 2 is a view depicting an example of a usage scene of a robot.

FIG. 3 is a view depicting an example of an appearance shape of an HMD.

FIG. 4 is a view depicting functional blocks of the HMD.

FIG. 5 is a view depicting an appearance configuration of the robot.

FIG. 6 is a view depicting a configuration of an insertion member.

FIG. 7 depicts views illustrating a cross section of the robot.

FIG. 8 depicts views illustrating an example of the posture of a housingof the robot.

FIG. 9 depicts views illustrating another example of the posture of thehousing of the robot.

FIG. 10 is a view depicting functional blocks of the robot.

FIG. 11 is a view depicting a circuit configuration of a phasedifference amplification apparatus provided in a sound processing unit.

FIG. 12 depicts views illustrating a phase difference between signalwaveforms.

FIG. 13 depicts views illustrating a principle of amplifying a phasedifference between input signal waveforms.

FIG. 14 is a view depicting functional blocks of the robot forimplementing an applied technology.

FIG. 15 is a view depicting functional blocks of a processing apparatus.

FIG. 16 is a view illustrating a whole sphere panorama image.

FIG. 17 is a view illustrating picked up image data recorded in an imagerecording unit.

FIG. 18 is a view depicting a relationship between a frame imagegenerated by an image generation unit and image data.

FIG. 19 depicts views illustrating a general structure of the inside ofthe housing.

FIG. 20 is a view depicting functional blocks of an input system.

FIG. 21 depicts views illustrating a position of a protective cover.

FIG. 22 is a view depicting an example of the retained substance of amotion table.

DESCRIPTION OF EMBODIMENT

FIG. 1 is a view depicting an example of a configuration of aninformation processing system 1 according to an embodiment. Theinformation processing system 1 includes a robot 10 and a head-mounteddisplay apparatus (HMD) 100 a user A wears on the head. The HMD 100includes a display panel 102 for both eyes, earphones 104 for both ears,and a microphone 106. While the earphones 104 are adopted as soundoutputting means, headphones having a shape in which they are placed onthe ears may be adopted. The HMD 100 is connected to a network 4 throughan access point (AP) 2. While the AP 2 has functions as a wirelessaccess point and a router and the HMD 100 is connected to the AP 2through a known wireless communication protocol, the HMD 100 may beconnected to the AP 2 through a cable.

The robot 10 includes an actuator apparatus 12 and a housing 20 actuatedby the actuator apparatus 12 such that the posture thereof can bechanged. In the housing 20, a right camera 14 a, a left camera 14 b, aright microphone 16 a, a left microphone 16 b, and a speaker 18 areincorporated. In the following description, where the right camera 14 aand the left camera 14 b are not specifically distinguished from eachother, each of them is referred to as “camera 14,” and where the rightmicrophone 16 a and the left microphone 16 b are not specificallydistinguished from each other, each of them is referred to as“microphone 16.” In the embodiment, the cameras 14 and the microphones16 are provided in the housing 20 that is actuated by the actuatorapparatus 12, the speaker 18 may be provided in a semispherical housing36 of the actuator apparatus 12. The robot 10 is coupled to the network4 through an access point (AP) 3. While the robot 10 is connected to theAP 3 through a known wireless communication protocol, the robot 10 maybe coupled to the AP 3 otherwise through a cable.

In the information processing system 1, the HMD 100 and the robot 10 areconnected for communication to each other through the network 4. It isto be noted that, where the HMD 100 and the robot 10 exist nearby toeach other, the HMD 100 and the robot 10 may be connected directly forcommunication by wireless communication or wired communication withoutthrough an AP. In the information processing system 1, the robot 10operates so to move as an avatar of the user A. A motion of the HMD 100worn by the user A is transmitted to the robot 10, and the actuatorapparatus 12 moves the housing 20 in an interlocking relationship withthe motion of the HMD 100. For example, if the user A shakes the head inthe forward or rearward direction, then the actuator apparatus 12 movesthe housing 20 so as to be shaken in the forward and rearward direction,and, if the user shakes the head in the leftward or rightward direction,then the actuator apparatus 12 moves the housing 20 so as to be shakenin the leftward or rightward direction. Consequently, a person aroundthe robot 10 can communicate with the user A with such a sense that theuser A exists at the site.

The right camera 14 a and the left camera 14 b are disposed on the frontface of the housing 20 in a predetermined spaced relationship from eachother in the horizontal direction. The right camera 14 a and the leftcamera 14 b configure a stereo camera, and the right camera 14 a picksup an image for the right eye in a predetermined cycle and the leftcamera 14 b picks up an image for the left eye in the predeterminedcycle. The picked up right eye image and left eye image are transmittedto the HMD 100 of the user A on the real time basis. The HMD 100displays the received right eye image on a display panel for the righteye and displays the received left eye image on a display panel for theleft eye. Consequently, the user A can watch a video in a direction inwhich the housing 20 of the robot 10 is directed on the real time basis.

The right microphone 16 a and the left microphone 16 b are disposed inthe housing 20 in a predetermined spaced relationship from each other inthe horizontal direction. The right microphone 16 a and the leftmicrophone 16 b configure a stereo microphone and are disposed in apredetermined spaced relationship from each other in the horizontaldirection such that periods of time until sound reaches differ dependingupon the position of a sound source. The difference between reachingtime periods of sound appear as a phase difference between sound signalsgenerated by the right microphone 16 a and the left microphone 16 b. Itis to be noted that, in order to increase the phase difference betweensound signals of the right microphone 16 a and the left microphone 16 b,it is preferable to dispose the right microphone 16 a and the leftmicrophone 16 b in a spaced relationship from each other by a distanceas great as possible, and particularly, on the opposite side faces ofthe housing 20.

Sound signals generated by the right microphone 16 a and the leftmicrophone 16 b are processed in such a manner as hereinafter describedand transmitted as sound data for the right ear and sound data for theleft ear to the HMD 100 of the user A on the real time basis. The HMD100 outputs the received right ear sound data from the earphone 104 forthe right ear and outputs the received left ear sound data from theearphone 104 for the left ear. Consequently, the user A can hear soundaround the robot 10 on the real time basis.

Although it is known that human beings sense the position of a soundsource in the leftward and rightward direction from the difference inreaching time period of sound waves to both ears, the position of thesound source is actually sensed depending not only upon the differencein reaching time period but also upon the shape of the auricle forcollecting sound waves, the shape of the external ear canal fortransmitting sound waves to the tympanum and so forth. Further, in thecase where a sound source exists on the right side or the left side withrespect to the front of a human, in order for sound waves to reach theauricle on the distant side in comparison with the auricle on the nearside, since the face of the human is positioned on a route, the reachingtime difference of sound waves becomes greater than the distancedifference from the sound source.

On the other hand, since the front face of the housing 20 has a flatshape and the microphones 16 do not have a shape corresponding to theauricle or the external ear canal, the sound reaching time differencesubstantially corresponds to the distance difference between the soundsource and both microphones. While, in the embodiment, the rightmicrophone 16 a and the left microphone 16 b are disposed on theopposite side faces of the housing 20 so as to be provided at positionsspaced by a distance as great as possible from each other, it has turnedout by an experiment by the inventor of the present application that,even if a sound signal generated by the right microphone 16 a and asound signal generated by the left microphone 16 b are amplified andoutput from the earphone for the right ear and the earphone for the leftear, respectively, the position of the sound source in the leftward andrightward direction cannot be sensed well.

In particular, it has been turned out by the experiment that, incomparison with sound that humans are accustomed to hear every day, thephase difference between sound signals generated by the right microphone16 a and the left microphone 16 b is so small as to perceive theleftward or rightward direction. Therefore, the robot 10 includes amechanism for providing sound data made closer to sound that can beheard with both ears of a human being by amplifying the phase differencebetween the sound signals of the right microphone 16 a and the leftmicrophone 16 b to the HMD 100. This mechanism is hereinafter described.

In the HMD 100, the microphone 106 generates a sound signal emitted bythe user A. The sound data by the user A is transmitted on the real timebasis to the robot 10, and the robot 10 outputs the received sound datato the speaker 18. Consequently, a person around the robot 10 can hearthe voice emitted by the user A on the real time basis.

In this manner, in the information processing system 1, the robot 10 isremotely controlled by the user A to reproduce a motion of the face orvoice of the user A, and the user A can enjoy an image or sound aroundthe robot through the HMD 100. Further, the user A and the person aroundthe robot 10 can communicate with each other on the real time basis.Such an information processing system 1 as described above is utilizedusefully in various environments.

FIG. 2 depicts an example of a usage scene of the robot 10. In thisexample, a meeting is held in a room and the robot 10 that is an avatarof the user A is disposed on a table. In this example, the robot 10 isdirected toward four members in front of the robot 10 and the cameras 14pick up an image of the four members in front of the robot 10 within anangle of view. The robot 10 transmits the picked up images of thecameras 14 to the HMD 100 of the user A on the real time basis. The userA participates in the meeting while watching the situation of the roomthrough the display panel 102 of the HMD 100 and, if the user A speaks,then the voice of the user A is transmitted to the robot 10 on the realtime basis and the robot 10 outputs the voice of the user A from thespeaker 18.

Further, as described above, the robot 10 transmits sound data in whichthe phase difference between the sound signals generated by the left andright microphones 16 is amplified to the HMD 100 on the real time basis.Consequently, the user A can sense whether a person who speaks in theroom is positioned on the right side, on the left side, or in front ofthe housing 20 with respect to the direction in which the housing 20 isdirected. If the user A senses that a person on the right side withrespect to the user A itself speaks, then the user A would turn the headto the right to face the right side. At this time, since also thehousing 20 of the robot 10 faces the right side in an interlockingrelationship with the motion of the head of the user A, the cameras 14pick up an image of the participant sitting on the right side.

In this manner, since the robot 10 that is an avatar of the user Ainterlocks with a motion of the user A, the user A can participate inthe meeting with such a sense that the user exists in the room while theuser A is at a distant place. Further, also participants actuallyexisting in the room can communicate with the user A without a sense ofincongruity from the voice of the user A or a motion of the housing 20.It is to be noted that the usage scene depicted in FIG. 2 is an example,and, also in other usage scenes, the user A can acquire viewing datafrom the robot 10 while the user A exists at a distant place.

FIG. 3 depicts an example of an appearance shape of the HMD 100. In thisexample, the HMD 100 is configured from an output mechanism unit 110 anda mounting mechanism unit 112. The mounting mechanism unit 112 includesa mounting band 108 that extends, where it is worn by the user, aroundthe head to fix the HMD 100 to the head. The mounting band 108 isconfigured from a material or a structure that allows adjustment of themounting band 108 in accordance with the periphery of the head of theuser.

The output mechanism unit 110 includes a housing 114 having a shape forcovering the left and right eyes of a user in a state in which the userwears the HMD 100, and the display panel 102 is provided at a positionfacing the eyes in the inside of the housing 114. The display panel 102may be a liquid crystal panel, an organic electroluminescence (EL) panelor the like. In the inside of the housing 114, a pair of left and rightoptical lenses are provided which are positioned between the displaypanel 102 and the eyes of the user and increases the viewing angle ofthe user.

The HMD 100 further includes the earphones 104 that are to be insertedinto the ears of the user when the HMD 100 is worn. It is to be notedthat the earphones 104 are an example of sound outputting means, and theHMD 100 may include a headphone. At this time, the HMD 100 and theheadphone may be configured integrally or may be configured as separatemembers each other.

The HMD 100 transmits sensor information detected by a posture sensorand sound data obtained by encoding a sound signal from the microphone106 to the robot 10, and receives image data and sound data generated bythe robot 10 and outputs the received data from the display panel 102and the earphones 104.

It is to be noted that, while the HMD 100 depicted in FIG. 3 is animmersive type (non-transmission type) display apparatus that fullycovers both eyes, the HMD 100 may otherwise be a transmission typedisplay apparatus. Further, while the shape may be that of such a hattype as depicted in FIG. 3, it may be that of the glasses type. It is tobe noted that the HMD 100 may be configured not only from a headmounting display apparatus for exclusive use but also from a terminalapparatus that includes a display panel, a microphone, and a speaker anda housing that fixes the display panel of the terminal apparatus at aposition just in front of the user. The terminal apparatus may be anapparatus including a comparatively small display panel like, forexample, a smartphone or a portable game machine.

FIG. 4 depicts functional blocks of the HMD 100. A control unit 120 is amain processor that processes and outputs various signals such as animage signal, an audio signal, and sensor information, data, andcommands. A storage unit 122 temporarily stores data, commands and soforth to be processed by the control unit 120. A posture sensor 124detects posture information such as a rotational angle, a tilt and soforth of the HMD 100 in a predetermined cycle. The posture sensor 124 atleast includes a three-axis acceleration sensor and a three-axisgyroscopic sensor. The microphone 106 converts voice of the user into anelectric signal to generate a sound signal.

A communication controlling unit 126 transmits and receives signals anddata to and from the robot 10 by wired communication or wirelesscommunication through a network adapter or an antenna. The communicationcontrolling unit 126 receives posture information detected by theposture sensor 124 and sound data obtained by encoding a sound signalfrom the microphone 106 from the control unit 120 and transmits thereceived data to the robot 10. Further, the communication controllingunit 126 receives and supplies image data and sound data from the robot10 to the control unit 120. If image data and sound data are receivedfrom the robot 10, then the control unit 120 supplies the image data tothe display panel 102 so as to be displayed and supplies the sound datato the earphone 104 so as to be output as sound.

FIG. 5 depicts an appearance configuration of the robot 10. The housing20 accommodates the cameras 14, the microphones 16, and the speaker 18therein. The cameras 14 and the speaker 18 are provided on the frontface of the housing, and the microphones 16 are provided on side facesof the housing. The cameras 14, the microphones 16, and the speaker 18operate with power supplied thereto through a power line (not depicted)from a power supply apparatus accommodated in the housing 36.

The housing 20 has a protective cover 19 such that, in a state in whichthe robot 10 is not used, namely, in a state in which the power supplyto the robot 10 is turned off, the protective cover 19 is disposed at aclosing position at which it covers the front face of the housing toprotect the cameras 14 and the speaker 18. The protective cover 19 isattached such that it has pivots provided in an inwardly projectingmanner at the opposite ends in the longitudinal direction of theprotective cover 19 and inserted in a pair of pivot holes in the sidewalls of the housing. Consequently, the protective cover 19 is attachedfor pivotal motion around the axis of the pivots with respect to thehousing 20. In the state depicted in FIG. 5, the protective cover 19 isdisposed at an open position rotated approximately by 180 degrees fromthe closing position such that the cameras 14 are exposed and can pickup an image of the surroundings. The protective cover 19 preferably hasa stopper mechanism by which it is fixed at the open position.

It is to be noted that, in a state in which the robot 10 is used, theprotective cover 19 may be driven and controlled in response to theemotion or the tilt of the head of a user who wears the HMD 100. In thiscase, a motor serving as a driving unit is provided in the housing 20and can control the operation of the protective cover 19 by connectingthe motor shaft to the pivots of the protective cover 19. In this case,the stopper mechanism may not be provided, and the protective cover 19may be rotatable within a range of approximately 270 degrees from theclosing position.

The housing 20 is supported such that the posture thereof can be changedby the actuator apparatus 12. The actuator apparatus 12 includes a legunit 40, the semispherical housing 36 supported at an upper portion ofthe leg unit 40, and a driving mechanism 50 for driving the housing 20.The driving mechanism 50 includes a first arc-shaped arm 32 having afirst elongated through-hole 32 a formed in the longitudinal directionthereof, a second arc-shaped arm 34 having a second elongatedthrough-hole 34 a formed in the longitudinal direction thereof, and apedestal 30 that supports the first arc-shaped arm 32 and the secondarc-shaped arm 34 for pivotal motion in a state in which the firstarc-shaped arm 32 and the second arc-shaped arm 34 cross with eachother. The pedestal 30 is covered on the upper side thereof with a cover38, and in a space covered with the cover 38, motors for individuallypivoting the first arc-shaped arm 32 and the second arc-shaped arm 34are disposed. It is to be noted that the pedestal 30 is supported forpivotal motion with respect to the housing 36, and a motor for rotatingthe pedestal 30 is disposed in the housing 36.

The first arc-shaped arm 32 and the second arc-shaped arm 34 are formedin a semicircular shape and are supported at the opposite end portionsthereof on the pedestal 30 such that they have the same center ofrotation. The diameter of the semicircular first arc-shaped arm 32 is alittle greater than the diameter of the semicircular second arc-shapedarm 34, and the first arc-shaped arm 32 is disposed on the outerperiphery side of the second arc-shaped arm 34. The first arc-shaped arm32 and the second arc-shaped arm 34 may be disposed so as to beorthogonal to each other on the pedestal 30. In the embodiment, a lineinterconnecting the opposite end portions of the first arc-shaped arm 32supported on the pedestal 30 and a line interconnecting the opposite endportions of the second arc-shaped arm 34 supported on the pedestal 30are orthogonal to each other. An insertion member 42 is inserted in thefirst elongated through-hole 32 a and the second elongated through-hole34 a and disposed at the crossing position of the first elongatedthrough-hole 32 a and the second elongated through-hole 34 a. Theinsertion member 42 slidably moves in the first elongated through-hole32 a and the second elongated through-hole 34 a by pivotal motion of thefirst arc-shaped arm 32 and the second arc-shaped arm 34.

FIG. 6 depicts a configuration of the insertion member 42. The insertionmember 42 includes a first restriction portion 42 a having a widthgreater than that of the first elongated through-hole 32 a and a secondrestriction portion 42 b having a width greater than the secondelongated through-hole 34 a such that it maintains the insertion statethereof in the first elongated through-hole 32 a and the secondelongated through-hole 34 a. The first restriction portion 42 a isdisposed on the upper side with respect to the first elongatedthrough-hole 32 a while the second restriction portion 42 b is disposedon the lower side with respect to the second elongated through-hole 34 ato prevent the insertion member 42 from dropping out of the firstelongated through-hole 32 a and the second elongated through-hole 34 a.The insertion member 42 may be structured such that, when the insertionmember 42 is to be attached to the first elongated through-hole 32 a andthe second elongated through-hole 34 a, one of the first restrictionportion 42 a and the second restriction portion 42 b is formed as aseparate member from a stem portion 42 c, and in a state in which thestem portion 42 c is inserted in the first elongated through-hole 32 aand the second elongated through-hole 34 a, the first restrictionportion 42 a or the second restriction portion 42 b is fixed to an endportion of the stem portion 42 c.

The stem portion 42 c is a portion to be inserted in the first elongatedthrough-hole 32 a and the second elongated through-hole 34 a and isnormally positioned at the crossing location of the first elongatedthrough-hole 32 a and the second elongated through-hole 34 a. The stemportion 42 c is restricted against rotation in the first elongatedthrough-hole 32 a and in the second elongated through-hole 34 a. In theembodiment, the stem portion 42 c has a rectangular cross section havinga width a little greater than the width of the first elongatedthrough-hole 32 a and the second elongated through-hole 34 a such that,although rotation of the stem portion 42 c is restricted in the firstelongated through-hole 32 a and in the second elongated through-hole 34a, rotation of the stem portion 42 c may be restricted by some othermeans. For example, a rail may be provided on an inner circumferentialface of the second arc-shaped arm 34 while a rail groove is provided onthe second restriction portion 42 b such that rotation of the stemportion 42 c is restricted through fitting engagement between the railand the rail groove. The housing 20 is attached to the first restrictionportion 42 a, and since rotation of the stem portion 42 c is restricted,the housing 20 can be maintained in a desired posture.

It is to be noted that the stem portion 42 c is slidably movable in thefirst elongated through-hole 32 a and in the second elongatedthrough-hole 34 a because it has a width smaller than the width of thefirst elongated through-hole 32 a and the second elongated through-hole34 a. Consequently, the insertion member 42 can move along the firstelongated through-hole 32 a and can move along the second elongatedthrough-hole 34 a by rotation of the first arc-shaped arm 32 and thesecond arc-shaped arm 34, respectively.

FIG. 7 depicts a cross section of the robot 10. FIG. 7 mainly depicts adriving system of the robot 10 and omits illustration of a controlcircuit board, a memory, a wiring line and so forth. FIG. 7(a) depicts across section taken along the second arc-shaped arm 34 in a state inwhich the first arc-shaped arm 32 and the second arc-shaped arm 34 areerected uprightly by 90 degrees with respect to the pedestal 30, andFIG. 7(b) depicts a cross section taken along the first arc-shaped arm32 in the state in which the first arc-shaped arm 32 and the secondarc-shaped arm 34 are erected uprightly by 90 degrees with respect tothe pedestal 30.

A first motor 52 is provided for rotating the first arc-shaped arm 32,and a second motor 54 is provided for rotating the second arc-shaped arm34. The first motor 52 and the second motor 54 are disposed on thepedestal 30 such that, when the pedestal 30 rotates, also the firstmotor 52 and the second motor 54 rotate together with the pedestal 30. Athird motor 56 is provided for rotating the pedestal 30 and is disposedin the housing 36. The first motor 52, the second motor 54, and thethird motor 56 are rotated by power supplied from a power supplyapparatus not depicted.

Since the first motor 52 rotates the first arc-shaped arm 32 and thesecond motor 54 rotates the second arc-shaped arm 34 and besides thethird motor 56 rotates the pedestal 30, the actuator apparatus 12 canchange the direction and the posture of the housing 20 attached to theinsertion member 42.

FIGS. 8 and 9 are views depicting examples of the posture of the housing20 of the robot 10.

FIG. 8(a) and FIG. 8(b) depict an example in which the housing 20 istilted in the leftward or rightward direction. FIG. 9(a) and FIG. 9(b)depict an example in which the housing 20 is tilted in the forward orrearward direction. The driving mechanism 50 of the robot 10 can causethe housing 20 to take an arbitrary posture. The posture of the housing20 is controlled by adjustment of the driving amount of the first motor52 and the second motor 54, and the direction of the housing 20 iscontrolled by adjusting the driving amount of the third motor 56.

FIG. 10 depicts functional blocks of the robot 10. The robot 10 includesan input system 22 that accepts and processes an input from the outside,and an output system 24 that processes outputting to the outside. Theinput system 22 includes a reception unit 60, a sensor informationacquisition unit 62, a motion detection unit 64, a gaze directiondetermination unit 66, an actuator controlling unit 68, a sound dataacquisition unit 70, and a sound processing unit 72. Meanwhile, theoutput system 24 includes an image processing unit 80, a soundprocessing unit 82, and a transmission unit 90.

The elements indicated as functional blocks that perform variousprocesses in FIG. 10 can be configured, in hardware, from a circuitblock, a memory, and some other large scale integrations (LSIs) and areimplemented, in software, by a program loaded in the memory and soforth. Accordingly, it is recognized by those skilled in the art thatthe functional blocks can be implemented in various forms only fromhardware, only from software, or from a combination of hardware andsoftware, and they are not restricted to any of them.

As described hereinabove, the HMD 100 transmits sensor informationdetected by the posture sensor 124 and sound data obtained by encoding asound signal generated by the microphone 106, and the reception unit 60receives the sensor information and the sound data. The sound dataacquisition unit 70 acquires the received sound data, and the soundprocessing unit 72 carries out a sound process and outputs the sounddata from the speaker 18. Consequently, the robot 10 can reproduce thevoice of the user A on the real time basis, and a person around therobot 10 can hear the voice of the user A.

The sensor information acquisition unit 62 acquires posture informationdetected by the posture sensor 124 of the HMD 100. The motion detectionunit 64 detects the posture of the HMD 100 worn on the head of the userA. The gaze direction determination unit 66 determines the gazedirection of the cameras 14 of the housing 20 in response to the postureof the HMD 100 detected by the motion detection unit 64.

The motion detection unit 64 performs a head tracking process fordetecting the posture of the head of the user on which the HMD 100 isworn. The head tracking process is performed in order to cause the fieldof view, which is to be displayed on the display panel 102 of the HMD100, to interlock with the posture of the head of the user, and by thehead tracking process of the embodiment, the rotational angle withrespect to a horizontal reference direction and the tilt angle withrespect to a horizontal plane of the HMD 100 are detected. Thehorizontal reference direction may be set as a direction in which theHMD 100 is directed, for example, when the power supply to the HMD 100is turned on.

The gaze direction determination unit 66 determines the gaze directionin response to the posture of the HMD 100 detected by the motiondetection unit 64. This gaze direction is a gaze direction of the user Aand hence is a gaze direction (optical axis direction) of the cameras 14of the robot 10 that is an avatar of the user A.

In order to cause the gaze direction (optical axis direction) of thecameras 14 to interlock with the gaze direction of the user A, it isnecessary to set a reference posture of the robot 10 in advance. WhileFIG. 5 depicts a state in which the first arc-shaped arm 32 and thesecond arc-shaped arm 34 are erected uprightly by 90 degrees withrespect to the pedestal 30, this state may be set as a horizontaldirection while the direction in which the front face of the housing 20is directed when the power supply to the robot 10 is turned on is set asthe horizontal reference direction. It is to be noted that the robot 10may have a posture sensor similarly to the HMD 100 such that thehorizontal direction can be set autonomously.

In a state in which the reference posture of the HMD 100 and the robot10 is set, the gaze direction determination unit 66 may determine therotational angle and the tilt angle detected by the motion detectionunit 64 as they are as the gaze direction (optical axis direction) ofthe cameras 14. When the motion detection unit 64 detects the rotationalangle and the tilt angle of the HMD 100, the gaze directiondetermination unit 66 determines the gaze direction of the HMD 100 as avector (x, y, z) of a three-dimensional coordinate system, and at thistime, the gaze direction determination unit 66 may determine the gazedirection of the cameras 14 of the robot 10 as same (x, y, z) or maydetermine (x′, y′, z′) that is a form of (x, y, z) to which somecorrection is applied.

The actuator controlling unit 68 controls the direction of the cameras14 such that it becomes the gaze direction determined by the gazedirection determination unit 66. In particular, the actuator controllingunit 68 adjusts the power to be supplied to the first motor 52, thesecond motor 54, and the third motor 56 such that the motion of thehousing 20 follows up the motion of the HMD 100. The motor drivingcontrol by the actuator controlling unit 68 is carried out on the realtime basis, and accordingly, the direction of the housing 20 is moved ina similar manner to the direction of the line of sight of the user A.

According to the actuator apparatus 12 of the embodiment, although thehousing 20 is driven with reference to the center of pivotal motion ofthe first arc-shaped arm 32 and the second arc-shaped arm 34, thismotion indicates a motion same as that of the head of a person. Theactuator apparatus 12 reproduces the motion of the head of the user A bysuch a simple structure that two semicircular arms cross with eachother.

A person transmits its intention by a motion of the head. For example,although, in Japan, if the head is shaken vertically, then thisrepresents the affirmation but if the head is shaken sideways, then thisrepresents the negation, since the actuator apparatus 12 moves thehousing 20 similarly to a motion of the head of the user A, a personaround the robot 10 can feel the intention of the user A from the motionof the housing 20. Therefore, that the motion of the head of the user Acan be reproduced by a simple structure is very useful in thetele-existence technology.

Now, the output system 24 is described.

In the output system 24, the right camera 14 a and the left camera 14 bare directed in directions controlled by the actuator apparatus 12 andpick up images within individual angles of view. The right camera 14 aand the left camera 14 b may be disposed in a spaced relationship fromeach other such that the distance therebetween becomes equal to anaverage distance between both eyes of an adult. Right eye image datapicked up by the right camera 14 a and left eye image data picked up bythe left camera 14 b are transmitted from the transmission unit 90 tothe HMD 100, in which they are displayed in the right half and the lefthalf of the display panel 102, respectively. Those images form aparallax image as viewed from the right eye and the left eye and aredisplayed in the two divisional regions of the display panel 102, bywhich a stereoscopic image can be formed. It is to be noted that, sincethe user A views the display panel 102 through an optical lens, theimage processing unit 80 may generate image data whose opticaldistortion by the lenses is corrected in advance and supply the imagedata to the HMD 100.

The right camera 14 a and the left camera 14 b perform image pickup in apredetermined cycle (for example, in 1/60 second), and the transmissionunit 90 transmits image data without delay to the HMD 100. Consequently,the user A can view a surrounding situation of the robot 10 on the realtime basis, and can view a desired direction by changing the directionof the face.

The right microphone 16 a and the left microphone 16 b convert soundaround the robot 10 into an electric signal to generate sound signals.In the following description, the sound signal generated by the rightmicrophone 16 a is referred to as “first sound signal,” and the soundsignal generated by the left microphone 16 b is referred to as “secondsound signal.” Since the right microphone 16 a and the left microphone16 b are disposed in a spaced relationship from each other in thehorizontal direction on the housing 20 as described hereinabove, a phasedifference appears between the first sound signal generated by the rightmicrophone 16 a and the second sound signal generated by the leftmicrophone 16 b.

The inventor of the present application obtained by an experiment that,in the case where the first sound signal and the second sound signal areencoded keeping the phase difference between them as it is and thenprovided to the HMD 100, the user cannot recognize the direction of thesound source, namely, the user is difficult to discriminate whether thesound is heard from the right side or the left side. While, in theexperiment, the width of the housing 20 in the horizontal direction wasset approximately to the width of the face of an adult human (16 cm),since the sound wave transmission structure of the ears of a human beingcannot be reproduced by the microphones 16, the conclusion was obtainedthat only the phase difference between the first sound signal and thesecond sound signal is insufficient for a human being to perceive thedirection of the sound source.

As means for solving this, it seems recommendable to increase the widthof the housing 20 in the horizontal direction, and to increase the phasedifference between the first sound signal and the second sound signal.However, in this case, the weight of the housing 20 increases, and itbecomes necessary to increase the output power of the motors used in theactuator apparatus 12. Further, if the width of the housing 20 in thehorizontal direction is increased, then since the distance between theright microphone 16 a and the left microphone 16 b becomes greater thanthe distance between both ears of a human being, sound signals thatprovide a feeling different from the feeling when a human being actuallyhears sound are acquired.

Therefore, the inventor of the present application has figured out tosolve this problem by amplifying the phase difference between the firstsound signal and the second sound signal. The sound processing unit 82has a function for amplifying the phase difference between the firstsound signal generated by the right microphone 16 a and the second soundsignal generated by the left microphone 16 b as hereinafter described.It is to be noted that, since it is necessary for the robot 10 totransmit microphone sound on the real time basis to the HMD 100, thesound processing unit 82 implements the phase difference amplificationfunction by a hardware circuit.

FIG. 11 depicts a circuit configuration of a phase differenceamplification apparatus 82 a provided in the sound processing unit 82.The phase difference amplification apparatus 82 a is an analog circuitapparatus that amplifies and outputs the phase difference between afirst sound signal v_(R) generated by the right microphone 16 a and asecond sound signal v_(L) generated by the left microphone 16 b.

If the first sound signal v_(R) is input from the right microphone 16 a,then a first amplifier 84 a outputs a first positive phase signal V_(R)⁺ obtained by amplifying the first sound signal v_(R) and a firstreverse phase signal V_(R) ⁻ obtained by inverting and amplifying thefirst sound signal v_(R). Although the first amplifier 84 a may beconfigured from an operational amplifier that amplifies and outputs apositive-phase component of an input signal and another operationalamplifier that amplifies and outputs a reverse-phase component of theinput signal, it may otherwise be configured from an operationalamplifier having two output terminals that output a positive-phasecomponent and a reverse-phase component.

Meanwhile, if the second sound signal v_(L) is input from the leftmicrophone 16 b, then a second amplifier 84 b outputs a second positivephase signal V_(L) ⁺ obtained by amplifying the second sound signalv_(L) and a second reverse phase signal V_(L) ⁻ obtained by invertingand amplifying the second sound signal v_(L). Similarly to the firstamplifier 84 a, the second amplifier 84 b may be configured from twooperational amplifiers that individually output a positive-phasecomponent and a reverse-phase component or may otherwise be configuredfrom a single operational amplifier that output both a positive-phasecomponent and a reverse-phase component.

The first adder 86 a outputs an output signal V_(rOUT) obtained byadding a signal obtained by multiplying the first positive phase signalV_(R) ⁺ by a first coefficient (by ∝) and another signal obtained bymultiplying the second reverse phase signal V_(L) ⁻ by a secondcoefficient (by β). Here, ∝ and β indicate values higher than 0 butequal to or lower than 1. It is to be noted that ∝ and β are set so asto different from each other, and in this example, ∝>β. The outputsignal V_(rOUT) is represented by the following expression.

V _(rOUT) =∝×V _(R) ⁺ +β×V _(L) ⁻

Although the first adder 86 a may be an addition circuit that adds anoutput of a voltage dividing circuit that divides the first positivephase signal V_(R) ⁺ to ∝ times and an output of another voltagedividing circuit that divides the second reverse phase signal V_(L) ⁻ toβ times, it may otherwise be an operational amplifier that adds avoltage signal obtained by multiplying the first positive phase signalV_(R) ⁺ by ∝ and another voltage signal obtained by multiplying thesecond reverse phase signal V_(L) ⁻ by β.

The second adder 86 b outputs a output signal V_(lOUT) obtained byadding a signal obtained by multiplying the second positive phase signalV_(L) ⁺ by the first coefficient (by ∝) and another signal obtained bymultiplying the first reverse phase signal V_(R) ⁻ by the secondcoefficient (by β). The output signal V_(lOUT) is represented by thefollowing expression.

V _(lOUT) =∝×V _(L) ⁺ +β×V _(R) ⁻

Although the second adder 86 b may be an addition circuit that adds anoutput of a voltage dividing circuit that divides the second positivephase signal V_(L) ⁺ to ∝ times and an output of another voltagedividing circuit that divides the first reverse phase signal V_(R) ⁻ toβ times, it may otherwise be an operational amplifier that adds avoltage signal obtained by multiplying the second positive phase signalV_(L) ⁺ by ∝ and another voltage signal obtained by multiplying thefirst reverse phase signal V_(R) ⁻ by β.

A third amplifier 88 a multiplies the output signal V_(rOUT) of thefirst adder 86 a by a third coefficient (by γ) and outputs V_(ROUT), anda fourth amplifier 88 b multiplies the output signal V_(lOUT) of thesecond adder 86 b by the third coefficient (by γ) and outputs V_(LOUT).In the sound processing unit 82, the output signals V_(ROUT) andV_(LOUT) from the phase difference amplification apparatus 82 a areindividually speech coded and transmitted as right ear sound data andleft ear sound data from the transmission unit 90 to the HMD 100.

FIG. 12 depicts views illustrating a phase difference between signalwaveforms. FIG. 12(a) depicts a relationship in waveform between thefirst sound signal v_(R) generated by the right microphone 16 a and thesecond sound signal v_(L) generated by the left microphone 16 b. Here,for the convenience of description, a relationship between the firstpositive phase signal V_(R) ⁺ and the second positive phase signal V_(L)⁺ obtained by amplifying the first sound signal v_(R) and the secondsound signal v_(L), respectively, to an equal number of times. In theinput waveforms, the sound source is disposed on the right side asviewed from the housing 20 of the robot 10, and the phase of the firstpositive phase signal V_(R) ⁺ is advanced a little from that of thesecond positive phase signal V_(L) ⁺ and the amplitude is higher withthe first positive phase signal V_(R) ⁺.

FIG. 12(b) depicts a relationship in waveform between the output signalV_(rOUT) of the first adder 86 a and the output signal V_(lOUT) of thesecond adder 86 b. If the phase difference between them is compared withthe phase difference between the input waveforms depicted in FIG. 12(a),then it is recognized that the phase difference between the outputwaveforms of the adders depicted in FIG. 12(b) is increased (amplified).

FIG. 13 depicts views illustrating a principle in amplifying a phasedifference between input signal waveforms. FIG. 13(a) represents thefirst positive phase signal V_(R) ⁺ and the first reverse phase signalV_(R) ⁻, and the second positive phase signal V_(L) ⁺ and the secondreverse phase signal V_(L) ⁻ in a two-dimensional coordinate system. Thephase difference between the first positive phase signal V_(R) ⁺ and thesecond positive phase signal V_(L) ⁺ is θ.

FIG. 13(b) depicts the output signal V_(rOUT) of the first adder 86 aand the output signal V_(lOUT) of the second adder 86 b. As describedhereinabove, V_(rOUT) and V_(lOUT) are represented by the followingexpression.

V _(rOUT) =∝×V _(R) ⁺ +β×V _(L) ⁻

V _(lOUT) =∝×V _(L) ⁺ +β×v _(R) ⁻

In FIG. 13(b), ∝=1.0 and β=0.6 are set.

As depicted in FIG. 13(b), the phase difference between V_(rOUT) andV_(lOUT) becomes θ′ and is greater than the phase difference θ depictedin FIG. 13(a). In this manner, the phase difference amplificationapparatus 82 a amplifies the phase difference between two input soundsignals.

As a result of a simulation by the inventor of the present application,it has been found that, when the phase difference between the inputsignals is 15 degrees, the phase difference between the output signalsbecomes four times, namely, 60 degrees; when the phase differencebetween the input signals is 30 degrees, the phase difference betweenthe output signals becomes three times, namely, 90 degrees; and when thephase difference between the input signals is 45 degrees, the phasedifference between the output signals becomes approximately 2.7 times,namely, 120 degrees.

According to this simulation result, as the phase difference decreases,the amplification factor increases. In the actual housing 20, the phasedifference between the input signals is approximately 5 to 20 degrees,and since the amplification factor can be made great within this range,the phase difference amplification apparatus 82 a can increase the phasedifference between the output signals to such a degree that the user candistinguish the direction of the sound source. The output signalsV_(ROUT) and V_(LOUT) from the phase difference amplification apparatus82 a are individually speech coded and transmitted as right ear sounddata and left eye sound data from the transmission unit 90 to the HMD100.

In the HMD 100, the right ear sound data is output as sound from theearphone 104 for the right ear, and the left ear sound data is output assound from the earphone 104 for the left ear. The user A would recognizethe direction of the sound source by hearing the sounds, between whichthe phase difference is amplified, by both ears. If the user A feelsthat the voice is coming from the right side, then the user A would turnthe face to the right side. At this time, since the housing 20 of therobot 10 is directed to the right side in an interlocking relationshipwith the motion of the face of the user A (refer to FIG. 2), the cameras14 of the robot 10 pick up an image of the environment on the right sideand transmit the picked up image data on the real time basis to the HMD100. Consequently, the user A can talk while looking at the face of theuttering person, and an unprecedented superior user interface can beimplemented.

It is to be noted that, while, in the example described above, ∝ and βare set to ∝=1.0 and β=0.6, respectively, the values of ∝ and β arepreferably set appropriately by an experiment. As depicted in FIG. 5,the right microphone 16 a and the left microphone 16 b are provided atpositions at which the side faces of the housing 20 are depressed andwhich are positions on the farther side as viewed from the front face.Since the transmission structure of sound waves in the microphones 16depends upon the shape of the side faces of the housing, the ratio of ∝and β is preferably determined optimally by an experiment.

It is to be noted that, in FIG. 5, the microphones 16 are disposed onthe inner side in the horizontal direction of a rear plate 17. This isbecause it is intended to provide a role for making the frequencycharacteristic different between a sound wave from the front and a soundwave from the rear to reduce high frequency components from the rear. Inparticular, the rear plate 17 has such a function as an auricle of aperson with respect to the microphones 16 such that sound waves from therear wrap around the rear plate 17 to reach the microphones 16. It is tobe noted that, in order to make the frequency characteristics of soundwaves from the front and sound waves from the rear different from eachother, the rear plate 17 may further be formed so as to be expanded inthe upward and downward direction and the horizontal direction. Byforming such a sound wave blocking member like the rear plate 17 behindthe microphones 16, also it becomes possible for the user A todistinguish the position of the sound source in the forward and rearwarddirection.

In this manner, in the information processing system 1, the user A cantake communication freely with people around the robot 10 on the realtime basis using the robot 10 that is an avatar of the user A itself. Inthe following, a technology for further enhancing the availability ofthe information processing system 1 is proposed.

In the past, a technology of stitching (sewing) images picked up whilesuccessively changing the tilt of a camera to generate a whole spherepanorama image has been known. Recently, also pan/tilt cameras forexclusive use are sold, and even an individual can pick up a wholesphere panorama image.

In the information processing system 1, the robot 10 picks up an imageof surroundings directing the cameras 14 to a gaze direction accordingto a motion of the head of the user A. Where the user A faces variousdirections, the cameras 14 pick up an image in various directions. Byadding a three-dimensional vector representative of a gaze direction toeach picked up image, it is possible to generate a virtual whole spherepanorama image.

FIG. 14 depicts a modification to the functional blocks of the robot 10.The functional blocks in the modification assume the functional blocksdepicted in FIG. 10 and indicate that a determined gaze direction issupplied from the gaze direction determination unit 66 to the imageprocessing unit 80 in the functional blocks.

During use of the robot 10 by the user A, the transmission unit 90transmits image data for both eyes and sound data for both ears (in thefollowing, they are sometimes referred to collectively as “viewingdata”) to the HMD 100 of the user A through the network 4. At this time,the transmission unit 90 transmits the same viewing data also to aprocessing apparatus 200 through a router 5 via the network 4, and theprocessing apparatus 200 records the viewing data of the user A.

The processing apparatus 200 has a function for generating, whilerecording the viewing data of the user A, a whole sphere panorama imageon the real time basis on the basis of the image data of the user A andproviding an image according to the gaze direction of a user B differentfrom the user A to an HMD 100 a of the user B. It is to be noted thatthe HMD 100 a has a configuration same as that of the HMD 100 describedhereinabove. Although the processing apparatus 200 may be configured,for example, from a single server, it may otherwise be configured from aserver group that provides cloud services.

In order to make it possible for the processing apparatus 200 togenerate a whole sphere panorama image, the image processing unit 80adds, to each frame image data, vector information indicative of a gazedirection supplied from the gaze direction determination unit 66 andimage pickup time information indicative of an elapsed time period fromthe start point of image pickup. The vector information indicates thegaze direction of the cameras 14 of the robot 10. The image pickup timeinformation may be any information if it can represent time from thestart point of image pickup and may be, for example, a frame numberindicative of an order number of image pickup.

According to this technology, during use of the robot 10 by the user A,the user B would wear the HMD 100 a, and image data and sound datagenerated on the basis of viewing data of the user A supplied from therobot 10 are provided to the HMD 100 a. If the received viewing data ofthe user A are merely reproduced as they are, then the processingapparatus 200 may only streaming distribute the received viewing data asthey are to the HMD 100 a of the user B. However, according to thistechnology, the processing apparatus 200 can re-construct an image basedon the gaze direction of the user B from the whole sphere panorama imageconfigured on the basis of the image data of the user A and provide theimage to the HMD 100 a of the user B. It is to be noted that the sounddata are streaming distributed to the HMD 100 a of the user B.

FIG. 15 depicts functional blocks of the processing apparatus 200. Theprocessing apparatus 200 includes a reception unit 202, a sensorinformation acquisition unit 204, a motion detection unit 206, a gazedirection determination unit 208, an image determination unit 210, asound determination unit 212, a viewing data provision unit 214, atransmission unit 216, and a recording unit 218. The recording unit 218includes an image recording unit 220 and a sound recording unit 222. Ifthe reception unit 202 receives viewing data transmitted from the robot10, then the image recording unit 220 successively records the receivedimage data, and the sound recording unit 222 successively records thereceived sound data. It is to be noted that the image data have vectorinformation and image pickup time information upon image pickup for eachframe image.

The user B would transmit a reproduction instruction of the viewing dataof the user A to the processing apparatus 200 through the HMD 100 a.When the processing apparatus 200 accepts the reproduction instruction,it starts a reproduction process of the viewing data. The sounddetermination unit 212 determines sound data to be provided to the userB, and immediately reads out the sound data recorded in the soundrecording unit 222 from the sound recording unit 222 and provides thesound data to the viewing data provision unit 214. In short, the sounddetermination unit 212 streaming distributes the sound data providedfrom the robot 10 to the HMD 100 a. Accordingly, the user B can hearsound same as the sound, which is being heard by the user A, from theearphone 104 of the HMD 100 a.

During a reproduction process by the processing apparatus 200, thereception unit 202 receives sensor information transmitted from the HMD100 a the user B wears, and the sensor information acquisition unit 204acquires the received sensor information. This sensor information isposture information of the posture of the HMD 100 a detected by theposture sensor 124. The motion detection unit 206 detects the posture ofthe HMD 100 a worn on the head of the user B. The gaze directiondetermination unit 208 determines a gaze direction of a virtual camerain the whole sphere panorama image in response to the posture of the HMD100 a detected by the motion detection unit 206. The image determinationunit 210 determines image data to be provided to the user B andsynthesizes an image picked up by the virtual camera directed in thedetermined gaze direction using a plurality of image data recorded inthe image recording unit 220 to generate image data.

The viewing data provision unit 214 provides viewing data including theimage data determined by the image determination unit 210 and the sounddata determined by the sound determination unit 212 from thetransmission unit 216 to the HMD 100 a of the user B.

The components indicated as functional blocks that perform variousprocesses in FIG. 15 can be configured, in hardware, circuit blocks, amemory, and other LSIs and are implemented in software from a programload in the memory and so forth. Accordingly, it is recognized by thoseskilled in the art that the functional blocks can be implemented invarious forms only from hardware, only from software, or from acombination of hardware and software, and they are not restricted to anyof them.

The processing apparatus 200 generates an omnidirectional panoramaimage. Accordingly, if the user B turns the head to the left or theright to turn the line of sight in the horizontal direction to the leftor the right, then a panorama image in the left direction or the rightdirection is displayed on the display panel 102 of the HMD 100 a.Further, if the user B tilts the head upwardly or downwardly to tilt theline of sight in the vertical direction, then a panorama image in theupward direction or the downward direction is displayed on the displaypanel 102 of the HMD 100 a.

FIG. 16 is a view illustrating a whole sphere panorama image generatedby the processing apparatus 200. According to this technology, a virtualenvironment is implemented in which an image to be viewed is changedwhen the user B positioned at the center of a sphere changes thedirection of its line of sight. The image determination unit 210stitches (sews) image data recorded in the image recording unit 220 togenerate a whole sphere panorama image.

In the embodiment, in order to simplify the description, the robot 10does not perform zooming of the cameras 14 and acquires image data undera fixed magnification ratio. Therefore, the image determination unit 210pastes the image data to the inner circumferential face of the wholesphere on the basis of the vector information added to the image datathereby to configure a whole sphere panorama image. It is to be notedthat a region in which a plurality of image data overlap with each otheris overwritten with the latest image data, and a whole sphere panoramaimage close to a real time situation can be configured thereby.

It is to be noted that the actual image generation process of the imagedetermination unit 210 is a process not for always configuring a wholesphere panorama image but for dynamically generating a frame image 7picked up from a center point 9 at which the user B is positioned inorder to reduce the processing load. At this time, the imagedetermination unit 210 preferably sets the image pickup range (angle ofview) of a virtual camera 8 so as to correspond to the actual imagepickup range (angle of view) of the cameras 14 of the robot 10. Thismakes it possible for the user B to see, at a timing at which the gazedirection of the user A and the gaze direction of the user B coincidewith each other, an image same as that viewed by the user A.

In this manner, the image determination unit 210 carries out an imagestitching process using vector information set as metadata in image datato generate a frame image 7 within an image pickup range determined fromthe gaze direction of the user B. The motion detection unit 206 performsa head tracking process for the user B to detect a rotational angle anda tilt of the head of the user B (actually, the HMD 100 a). Here, therotational angle of the HMD 100 a is a rotational angle with respect toa reference direction of a horizontal plane, and the reference directionmay be set, for example, as a direction in which the HMD 100 a isdirected when the power supply to the HMD 100 a is turned on. Meanwhile,the tilt of the HMD 100 a is a tilt angle with respect to the horizontalplane. For the head tracking process, a known technology may beutilized, and the motion detection unit 206 detects the rotational angleand the tilt of the HMD 100 a from sensor information detected by theposture sensor of the HMD 100 a.

The gaze direction determination unit 208 determines a posture of thevirtual camera 8 in a virtual sphere in accordance with the detectedrotational angle and tilt of the HMD 100 a. The virtual camera 8 isdisposed such that it picks up an image of the inner circumferentialface of the virtual sphere from the center point 9 of the virtualsphere, and the gaze direction determination unit 208 may determine thedirection of the optical axis of the virtual camera 8 so as to coincidewith the optical axis direction of the cameras 14 of the robot 10.

It has been described that, in the robot 10, after the gaze directiondetermination unit 66 determines the gaze direction of the HMD 100 ofthe user A as a vector (x, y, z) of a three-dimensional coordinatesystem, it may determine the gaze direction of the cameras 14 of therobot 10 as same (x, y, z). Also in the processing apparatus 200, if thegaze direction determination unit 208 determines the gaze direction ofthe HMD 100 a of the user B as a vector (x, y, z) of a three-dimensionalcoordinate system, then it may determine the gaze direction of thevirtual camera 8 as the same vector (x, y, z). Further, in the casewhere the gaze direction determination unit 66 corrects the gazedirection of the HMD 100 by a predetermined conversion expression todetermine the gaze direction of the cameras 14, also the gaze directiondetermination unit 208 may correct the gaze direction of the HMD 100 aby the same conversion expression to determine the gaze direction of thevirtual camera 8. By handling the individual three-dimensionalcoordinate systems in this manner, at a timing at which the gazedirection of the user A and the gaze direction of the user B coincidewith each other, the user B can see an image same as that seen by theuser A.

After the frame image 7 of the virtual camera 8 is generated, the imagedetermination unit 210 carries out optical distortion correction for theoptical lens and supplies the image data to the viewing data provisionunit 214. It is to be noted that, although one virtual camera 8 isdepicted in FIG. 16, actually two virtual cameras 8 for the left eye andthe right eye are disposed, and image data of them are generated on thebasis of left eye image data and right eye image data provided from therobot 10, respectively.

FIG. 17 is a view illustrating picked up image data recorded in theimage recording unit 220. Here, for the convenience of description, aplurality of image data for one eye are depicted, and image data forwhich affine transformation is carried out in regard to the gazedirection of the user B are disposed on a two-dimensional plane. It isto be noted that the gaze direction of the user B is hereinafterdescribed.

The image determination unit 210 has a function of joining overlappingportions of picked up images together to generate a whole spherepanorama image. For the technology for joining picked up imagestogether, a known technology may be utilized as described, for example,in Japanese Patent No. 5865388 by the same applicant. In the following,a method for selecting picked up image data from among a plurality ofpicked up image data recorded in the image recording unit 220 isdescribed.

In FIG. 17, five image data I1 to I5 are depicted. (x, y, z) included ineach image data represents a gaze direction (vector information) of thecamera 14 upon image pickup, and “t” represents image pickup timeinformation. Here, the image data I1 have vector information (x1, y1,z1) and image pickup time information t1 as additional information.Similarly, the image data I2 have vector information (x2, y2, z2) andimage pickup time information t2 as additional information. Similarly,the image data I3 have vector information (x3, y3, z3) and image pickuptime information t3 as additional information; the image data I4 havevector information (x4, y4, z4) and image pickup time information t4 asadditional information; and the image data I5 have vector information(x5, y5, z5) and image pickup time information t5 as additionalinformation.

It is to be noted that image pickup time information t1 to t5 that isadditional information represents elapsed time periods from the imagepickup start point (time 0) and has a relationship of t1<t2<t3<t4<t5.Accordingly, among the image data I1 to I5, the image data I1 are pickedup first, and the image data I5 are picked up last. The imagedetermination unit 210 selects image data for generating a syntheticimage on the basis of the image pickup time information and the gazedirection of the virtual camera 8 determined by the gaze directiondetermination unit 208.

In particular, the image determination unit 210 determines an imagepickup range (angle of view of the virtual camera 8) to be cut from thewhole sphere panorama image from the gaze direction of the virtualcamera 8 determined by the gaze direction determination unit 208,namely, from a direction in which the user B who wears the HMD 100 a isdirected, and extracts image data including the image included in theimage pickup range on the basis of the vector information added to theimage data.

FIG. 18 is a view depicting a relationship between a frame image 7 to begenerated by the image determination unit 210 and image data. Referringto FIGS. 17 and 18, the image data I1 to I5 are mapped on atwo-dimensional plane orthogonal to the gaze direction (X, Y, Z) of thevirtual camera 8 on the basis of the individual vector information, andthe position of each of the image data I1 to I5 is defined by fourvertex coordinates on the two-dimensional plane. The image determinationunit 210 determines the position of the angle of view (image pickuprange) of the virtual camera 8 on the whole sphere panorama image anddetermines the four vertex coordinates of the frame image 7 on thetwo-dimensional plane orthogonal to the gaze direction from the gazedirection (X, Y, Z) of the virtual camera 8. The image determinationunit 210 extracts image data included in the frame image 7 from amongthe image data recorded in the image recording unit 220. As depicted inFIG. 18, since the image data I1 to I5 include an image included in theframe image 7, they are extracted as image data included in the imagepickup range of the virtual camera 8.

The image determination unit 210 generates, for a region in which aplurality of image data overlap with each other, a synthetic imagepreferentially using image data having late image pickup timeinformation. In the example depicted in FIG. 18, the image data aresuccessively written into the frame image 7 beginning with image data ofearly image pickup time, namely, beginning with the image data I1 suchthat they are successively overwritten with newer image data tosynthesize the frame image 7.

In this manner, for a region in which a plurality of image data overlapwith each other, the image determination unit 210 generates a syntheticimage using image data having image pickup time information closer tothe current point of time. For example, in the case where a region inwhich the image data I4 and the image data I5 overlap with each otherexists in a region included in the image pickup range, the image data I5having later image pickup time are filled into the overlapping region.Consequently, a synthetic image can be generated using image data closerto the current point of time, and a synthetic image at time closer tothe current point of time can be provided to the user B.

In this image reproduction application, depending upon the direction inwhich the user B faces, the image data may be insufficient, resulting indifficulty to generate a frame image 7. Especially, immediately afterthe robot 10 starts image pickup, since the number of image data issmall in the first place, it sometimes occurs that the imagedetermination unit 210 cannot generate a frame image 7 according to thegaze direction of the user B. In the case where the user A does not movethe HMD 100 at all within an image pickup period, which cannot actuallyoccur, since the vector information of the image data recorded in theimage recording unit 220 is quite same, for example, if the user B facesin the just opposite direction to that of the user A, then image dataincluded in the image pickup range of the virtual camera 8 in the gazedirection does not exist.

In such a case as just described, the image determination unit 210 maygenerate image data by superimposing a message that an image in the gazedirection of the user B cannot be generated on the received image dataof the user A and provide the generated image data from the viewing dataprovision unit 214 to the HMD 100 a. For example, in the case where theimage in the gaze direction of the user B cannot be synthesized at aratio equal to or higher than a predetermined ratio (for example, 30%),the image determination unit 210 may not perform generation of asynthetic image and supply the image data viewed by the user A to theviewing data provision unit 214 together with the message describedabove.

Further, since the image determination unit 210 synthesizes a frameimage 7 from a plurality of image data, the generated frame image 7sometimes becomes a patch image and is degraded in visibility.Therefore, in the case where an image of a predetermined ratio (forexample, 50%) within an image pickup range cannot be formed from singleimage data, the image determination unit 210 may generate image data bysuperimposing a message that an image in the gaze direction of the userB cannot be generated on the image data of the user A.

In the example described above, it is described that the imagedetermination unit 210 generates a synthetic image preferentially usingimage data having later image pickup time information, in the case wherea predetermined ratio or more of a frame image 7 can be configured byusing image data having earlier image pickup time information, suchimage data having earlier image pickup time information may be used.

Further, as time passes, a variation occurs with the environment whoseimage is being picked up by the robot 10, and therefore, it may notpossibly be preferable to provide a synthetic image, for which imagedata in the past are used, to the user B. Therefore, the imagedetermination unit 210 may perform an image extraction process such thatimage data before a predetermined time period or more are not includedin the synthetic image.

The foregoing is directed to an example in which the user B utilizesviewing data of the user A on the real time basis. In the following, anapplied technology of this is described. In the applied technology, theprocessing apparatus 200 records viewing data not for the object of realtime reproduction of viewing data of the user A but for the object ofsecondary use of the viewing data.

For the object of secondary use of the viewing data, the imageprocessing unit 80 in the robot 10 adds image pickup time informationand vector information to each of the frame image data, and the soundprocessing unit 82 adds recording time information indicative of anelapsed time period from the recording start point to the sound data. Itis to be noted that, since image pickup (recording) by the cameras 14and sound recording through the microphones 16 are started at the sametiming, the image pickup start point and the sound recording start pointindicate the same timing. The image pickup time information and thesound recording time information may be time information generated by aclock generation unit of the robot 10. The form in which image data andsound data are added to additional information may be any form and maybe a form in which the processing apparatus 200 can refer to them whenit generates viewing data for reproduction.

In this applied technology, after the user A ends use of the robot 10, adifferent user B (who may be the user A) would wear the HMD 100 a andimage data and sound data generated on the basis of the viewing data ofthe user A recorded in the processing apparatus 200 are provided to theHMD 100 a. At this time, as described in connection with the embodiment,the processing apparatus 200 configures a whole sphere panorama image onthe basis of the viewing data of the user A and re-constructs an imageon the basis of the gaze direction of the user B from the whole spherepanorama image such that the image can be provided to the HMD 100 a ofthe user B. In this utilization environment, the robot 10 is not used.

Referring to FIG. 15, the image recording unit 220 records image datatransmitted from the robot 10 and the sound recording unit 222 recordssound data transmitted from the robot 10. In this applied technology,the image recording unit 220 and the sound recording unit 222 are in astate in which all of the viewing data transmitted from the robot 10 tothe user A are recorded already. It is to be noted that the image datahave image pickup time information and vector information upon imagepickup added thereto, and the sound data have recording time informationadded thereto.

The user B would issue a reproduction instruction of the viewing data ofthe user A to the processing apparatus 200 through the HMD 100 a. Whenthe processing apparatus 200 accepts the reproduction instruction, itstarts a reproduction process of the viewing data. It is to be notedthat, in the case where the recording unit 218 has viewing data for onehour recorded therein, the user B may start reproduction from anarbitrary point of time within the range of one hour. In this case, thereception unit 202 accepts a time designation from the user B andprovides the time designation to the image determination unit 210 andthe sound determination unit 212.

The sound determination unit 212 reads out sound data having recordingtime information corresponding to reproduction time informationindicative of an elapsed time period from the reproduction start pointfrom the sound recording unit 222 and provides the sound data to theviewing data provision unit 214. The reproduction start point signifiesa reproduction start point of the viewing data, and accordingly, theimage pickup start point and the sound recording start point indicatethe same timing. The sound determination unit 212 reads out sound datawhose sound recording time information coincides with the soundreproduction time information from the sound recording unit 222 andprovides the sound data to the viewing data provision unit 214.

During a reproduction process by the processing apparatus 200, thereception unit 202 receives sensor information transmitted from the HMD100 a the user B wears, and the sensor information acquisition unit 204acquires the received sensor information. This sensor information isposture information indicative of the posture of the HMD 100 a detectedby the posture sensor 124. The motion detection unit 206 detects theposture of the HMD 100 a worn on the head of the user B. The gazedirection determination unit 208 determines a gaze direction of thevirtual camera in response to the posture of the HMD 100 a detected bythe motion detection unit 206. The image determination unit 210synthesizes an image picked up by the virtual camera directed in thedetermined gaze direction using a plurality of image data recorded inthe image recording unit 220. The viewing data provision unit 214provides viewing data including the image data synthesized by the imagedetermination unit 210 and the sound data read out by the sounddetermination unit 212 from the transmission unit 216 to the HMD 100 a.

The image determination unit 210 stitches (sews) the image viewed by theuser A before the reproduction time of the viewing data by the user B todynamically generate a frame image 7 picked up from the center point 9at which the user B is positioned.

The image viewed by the user A before the reproduction time of theviewing data by the user B is described. In the case where image datafor one hour from the image pickup start point are recorded in the imagerecording unit 220, the reproduction time from the reproduction startpoint by the user B is specified by some timing within one hour. Forexample, in the case where the reproduction time is the timing at 15minutes after the reproduction start, images to which image pickup timeinformation within 15 minutes, namely, images picked up before 15minutes elapse from the image pickup start point, are views viewed bythe user A before the reproduction time. In short, if an image at thepoint of time of 15 minutes from the image pickup start is beingreproduced, then the image determination unit 210 generates a frameimage 7 using the image data to which the image pickup time informationwithin 15 minutes from the image pickup start is added, and if an imageat the point of time of 45 minutes from the reproduction start, then theimage determination unit 210 generates a frame image 7 using the imagedata to which the image pickup time information within 45 minutes fromthe image pickup start is added.

Referring to FIG. 18, the image determination unit 210 extracts imagedata to which image pickup time information before the reproduction timeinformation is added but does not extract image data to which imagepickup time information after the reproduction time information isadded. For example, if the time information for reproduction of a frameimage 7 is after time t3 but before time t4, then the imagedetermination unit 210 extracts the image data I1 to I3 but does notextract the image data I4 and I5. By generating a synthetic image usingimage data to which image pickup time information before thereproduction time information is added in this manner, the imagedetermination unit 210 does not allow the user B to view an image pickedup after the reproduction time.

Since the viewing data provision unit 214 is transmitting sound datahaving recording time information corresponding to reproduction time tothe HMD 100 a, the user B is hearing sound synchronized with thereproduction time. Therefore, the user B is generally aware of asituation before the reproduction time and can grasp, if the providedimage data are synthesized from image data before the reproduction time,what situation is being displayed. However, if the provided image dataare synthesized from image data after the reproduction time, then animage of which the user B is not aware is presented to the user B, andit is estimated that the user B is confused. Therefore, the imagedetermination unit 210 does not present an image picked up after thereproduction time to the user B.

It is to be noted that the image determination unit 210 generates asynthetic image using, for a region in which a plurality of image dataoverlap with each other, image data having image pickup time informationclose to the reproduction time information. For example, if a regionincluded in the image pickup range includes a region in which the imagedata I1 and the image data I2 overlap with each other, the image data I2picked up later than the image data I1 is filled into the overlappingregion. Consequently, it becomes possible to synthesize a syntheticimage using image data close to the reproduction time information, andan image synthesized from image data closest to the reproduction timecan be provided to the user B.

The present invention has been described with reference to theembodiment. The embodiment is exemplary, and it can be recognized bythose skilled in the art that various modifications can be made to thecomponents and the processes of the embodiment and that also suchmodifications are included in the scope of the present invention.

In the description of the embodiment, it is described that the imagedetermination unit 210 carries out an image stitching process togenerate a frame image 7 of an image pickup range determined from thegaze direction of the user B. In a modification, without carrying outthe image stitching process, the image determination unit 210 determinesimage data to be provided to the user B on the basis of the gazedirection of the virtual camera 8 and vector information added to imagedata recorded in the image recording unit 220.

In this modification, the image determination unit 210 determines theimage data to which the vector information corresponding to the gazedirection of the virtual camera 8 is added as image data to be providedto the user B. The vector information corresponding to the gazedirection of the virtual camera 8 includes vector information coincidentwith the gaze direction of the virtual camera 8 and vector informationthat can be regarded as substantially coincident with the gaze directionof the virtual camera 8. In particular, in the case where the gazedirection of the virtual camera 8 and the vector information are withina predetermined angle (for example, 10 degrees), the image determinationunit 210 may decide that the gaze direction of the virtual camera 8 andthe vector information substantially coincide with each other.

In the case where viewing data of the user A is to be synchronouslyreproduced, the image determination unit 210 determines, from among theimage data to which vector information corresponding to the gazedirection of the virtual camera 8 is added, image data having the latestimage pickup time information as image data to be provided to the userB. This makes it possible to provide an image closest to the currenttime to the user B.

In the case where image data to which vector information correspondingto the gaze direction of the virtual camera 8 is added is not recordedin the image recording unit 220, the image determination unit 210 maydetermine image data to which vector information that can be regardedsubstantially coincident in terms of the (x, y) components other thanthe component in the heightwise direction (z-axis direction) is added asimage data to be provided to the user B. The vector information that canbe regarded as coincident is vector information whose (x, y) componentsare within a range of a predetermined angle (for example, 7 degrees). Bychecking the approximation only of the (x, y) components, it becomeseasy for the image determination unit 210 to find out image data towhich vector information corresponding to the gaze direction of thevirtual camera 8 is added and avoid such a situation that image datacannot be provided to the user B.

In the case where viewing data of the user A are to be secondarily used,the image determination unit 210 determines image data to be provided tothe user B from images viewed by the user A before the reproduction timeof the viewing data by the user B. In short, the image determinationunit 210 determines, from among the image data to which image pickuptime information before the reproduction time information is added,image data to which vector information corresponding to the gazedirection of the virtual camera 8 is added. At this time, if a pluralityof image data are available, then the image determination unit 210preferably selects image data having image pickup time information closeto the reproduction time information.

It is to be noted that a case in which image data when the user A turnsthe head in the horizontal direction are recorded in the image recordingunit 220 is examined. If the user B turns the head in the oppositedirection to that of the user A with a slight delay from the user A,then such a situation that the image viewed by the user A is reverselyreproduced on the HMD 100 a may possibly occur. In this case, the timeseries of the image data is reversed, and there is the possibility thatthis may provide incompatibility. Therefore, in the case where, when theuser B continuously changes the gaze direction, image data of the user Acome to be reproduced reversely, the image determination unit 210 mayfix the image data such that image data to be provided are not changed.

In order to increase utility of the information processing system 1, therobot 10 may further include an input sensor for accepting an input fromthe outside such as a tactile sensor or a vibration sensor. In thefunctional blocks depicted in FIG. 10, the input sensor is provided inthe output system 24, and sensor information of the input sensor istransmitted from the transmission unit 90 to the HMD 100. The HMD 100may include outputting means for outputting sensor information such thatsensor information is converted into and transmitted as vibration to theuser A.

Further, while it is described that, in the information processingsystem 1, the robot 10 causes the housing 20 to interlock with a motionof the head of the user A, the robot 10 may further include means fortransmitting a state of the user A such as facial expressions. Forexample, the HMD 100 includes a sensor for detecting a motion of theeyes or the eyebrows of the user A who wears the HMD 100, means foranalyzing the tone of voice and so forth. The motion of the eyes or theeyebrows represents a facial expression or an emotion of the user, andthe tone of voice represents a psychological state or an emotion of theuser. The state information relating to the motion of the eyes or theeyebrows and/or the tone of voice is transmitted from the HMD 100 to therobot 10, and the robot 10 may drive a facial expression unit providedin the housing 20 to reproduce a facial expression or an emotion of theuser A. The facial expression unit may be a movable member (for example,a member that simulates the shape of an eyebrow) formed at an upperportion of each camera 14 on the front face of the housing 20, and themovable member is driven on the basis of information transmitted fromthe HMD 100. It is to be noted that the protective cover 19 attached forpivotal motion at an upper portion of the cameras 14 may be utilized asa facial expression unit simulating an eyebrow of a human being, or amotor for moving the protective cover 19 may be provided in the housing20 to move the protective cover 19. Alternatively, the facial expressionunit may be a display that represents a facial expression or apsychological state of the user A using a color such that a facialexpression or an emotion of the user A may be represented by changingthe display color.

In the following, the movable member provided on the robot 10 isdescribed. As described hereinabove, the movable member is provided inorder to create a facial expression of the robot 10. Although themovable member may be controlled in operation such that it represents afacial expression or an emotion of the user A who wears the HMD 100, itmay otherwise be controlled in operation on the basis of the posture ofthe housing 20. Since the robot 10 not only moves the housing 20 butalso moves the movable member attached to the housing 20, it providessuch a feeling to a person around the robot 10 that the robot 10 is theuser A.

As described so far, the housing 20 can be changed in posture by theactuator apparatus 12 and the actuator apparatus 12 causes the postureof the housing 20 to interlock with the motion of the HMD 100 worn bythe user A. The housing 20 and the actuator apparatus 12 configure therobot 10, and the housing 20 configures the face of the robot 10. Here,“to configure the face of the robot” signifies to form the housing 20such that, when a person around the robot 10 sees the robot 10, it canrecognize that the housing 20 is the face. The point in recognition thatthe housing 20 is the face is that the housing 20 includes the cameras14 positioned at an uppermost portion of the robot 10 and corresponds toan eye of a human being and that the posture of the housing 20interlocks with the motion of the head of the user, and the housing 20is recognized as the face of the robot 10 through the points.

FIG. 19(a) depicts a general structure of the front face of the housing20, and FIG. 19(b) depicts a general structure of the top face of thehousing 20. The housing 20 includes a front plate through which the lensof the cameras 14 is exposed, a bottom plate provided adjacent the frontplate and having the insertion member 42 attached thereto, the rearplate 17 provided adjacent the bottom plate and opposing to the frontplate, and a top plate provided adjacent the rear plate 17 and the frontplate and opposing to the bottom plate. The housing 20 further includesa pair of side plates that close up the opposite end sides of the frontplate, bottom plate, rear plate 17, and top plate. It is to be notedthat, while, in FIG. 9 and so forth, a manner in which the rear plate 17projects rearwardly is depicted, since the shape of the rear plate 17does not particularly have a relationship to the description givenbelow, in FIG. 19(a) and FIG. 19(b), the housing 20 is depicted in sucha manner that it has a parallelepiped elongated in the horizontaldirection. It is to be noted that the front face general structure ofFIG. 19(a) depicts a state in which the front plate is removed, and FIG.19(b) depicts a state in which the top plate is removed, and depictionof the speaker 18 and depiction of wiring lines including power linesand control signal lines for the cameras 14, the microphones 16, and adriving motor 142 are omitted.

In the inside of the housing 20, a structure for fixing various parts isprovided. Microphone accommodation portions 11 a and 11 b are providedon an inner wall of the side plates and form spaces for accommodatingthe right microphone 16 a and the left microphone 16 b therein,respectively. The right microphone 16 a and the left microphone 16 b areinserted in the microphone accommodation portions 11 a and 11 b from thefront and are fixed. The right camera 14 a and the left camera 14 b arefixed to camera supporting portions 13 provided in a projecting mannertoward the front from the inner wall of the rear plate 17 by screws 15.A motor fixing portion 140 is formed between the right camera 14 a andthe left camera 14 b, and the driving motor 142 is fixed to the motorfixing portion 140.

The protective cover 19 that is the movable member is a plate member ofa rectangular shape and is supported for movement on the housing 20. Theprotective cover 19 has pivots 19 a and 19 b provided in an inwardlyprojecting manner at the opposite ends thereof in the longitudinaldirection. Pivot holes are formed in the proximity of a front end atupper portions of the pair of side plates of the housing 20, and thepivot 19 a is inserted in the pivot hole of the right side plate whilethe pivot 19 b is inserted in the pivot hole of the left side plate suchthat the protective cover 19 is supported for pivotal motion on thehousing 20.

At least one of the pivot 19 a and the pivot 19 b is inserted in theinside of the housing farther than the inner wall of the side plate andis connected to a motor shaft 142 a of the driving motor 142 in theinside of the housing. In FIG. 19(a) and FIG. 19(b), both of the pivots19 a and 19 b extend to the inner side farther than the inner wall ofthe side plates of the housing, and the pivot 19 b from between thepivots 19 a and 19 b is fixed to one end of a transmission member 144.

The transmission member 144 is a member for transmitting rotation of thedriving motor 142 to the protective cover 19 that is the movable member.The transmission member 144 is fixed at one end thereof to the pivot 19b and at the other end thereof to the motor shaft 142 a of the drivingmotor 142. It is to be noted that, although the other end of thetransmission member 144 may be directly coupled to the motor shaft 142a, it may otherwise be fixed to an output power shaft of a speedreduction gear for reducing the speed of motor rotation. The protectivecover 19 is connected to the driving motor 142 by the transmissionmember 144 such that it is moved between the closing position at whichit protects the cameras 14 and a different position. While the powersupply to the robot 10 is turned off, the protective cover 19 protectsthe cameras 14, but while the power supply is turned on, the protectivecover 19 operates as the facial expression unit to implement effectiveuse of the protective cover 19.

Here, the transmission member 144 is formed as a member that deformstorsionally and elastically. In the present example, the transmissionmember 144 is a coil spring and delays rotational input power of themotor shaft 142 a in time by torsion of the coil spring to output therotational input to the pivot 19 b. It is to be noted that the torsionoccurring on the coil spring serves as restoring force, and after adelay from the rotational input of the motor shaft 142 a, the coilspring is rotated in the direction in which the torsion is to beeliminated.

Further, the transmission member 144 preferably has bending elasticityin the axial direction. Since various parts are accommodated in theinside of the housing 20 as depicted in FIG. 19(a) and FIG. 19(b), thespace in which the driving motor 142 is to be disposed is restricted.Since the transmission member 144 has bending elasticity in the axialdirection, even if the motor shaft 142 a and the pivot 19 b are notpositioned on the same axial line, they can be connected to each otherin a state in which the transmission member 144 is bent in the axialdirection. In FIG. 19(a) and FIG. 19(b), a manner is depicted in whichthe transmission member 144 couples the motor shaft 142 a and the pivot19 b to each other in a bent state such that axial misalignment betweenthe motor shaft 142 a and the pivot 19 b may be absorbed.

The reason why the transmission member 144 having torsional elasticityis adopted is that it is intended to delay rotation of the pivot 19 bfrom rotation of the motor shaft 142 a. Although the driving motor 142drives the protective cover 19 in response to the posture of the housing20, emotion information of a user and so forth as hereinafter described,the protective cover 19 becomes like an eyebrow positioned above theeyes (cameras 14) of the face of the robot. If the protective cover 19performs back and forth movement with flap at a high speed, then thismovement causes a surrounding person to feel bewildering, and besidessince the motion is different from a motion of the eyebrows of a humanbeing, this is not ideal as a motion of the facial expression unit.

Although rotation of the driving motor 142 is arithmetically operatedand controlled by software to perform a filtering process forrestricting the speed of rotation to a moderate one, it cannot beavoided that the load upon the arithmetic operation unit becomes heavy.Therefore, by transmitted rotation of the driving motor 142 to the pivot19 b of the protective cover 19 using the transmission member 144 whichdeforms torsionally and elastically, a high speed motion of theprotective cover 19 can be suppressed.

If the torsional rigidity of the transmission member 144 is high, thenthe transmission response delay to the pivot 19 b becomes less likely tooccur, and therefore, the torsional rigidity is preferably set low.However, the torsion amount actually occurring with the transmissionmember 144 relies also upon the weight of the protective cover 19.Therefore, in order to increase the response delay, the weight of theprotective cover 19 may be increased. Since coil springs having variouscharacteristics are sold at present, although it is advantageous fromthe point of view of cost to form the transmission member 144 as a coilspring, for example, the transmission member 144 may be a solid bar-likemember formed from a rubber material. By forming the transmission member144 from a rubber material, the transmission member 144 can be providedwith torsional elasticity and bending elasticity.

FIG. 20 depicts functional blocks of the input system 22 in the robot10. It is to be noted that the functional blocks depicted in FIG. 20 areadded to the input system 22 depicted in FIG. 10, and in FIG. 20, onlycomponents for driving and controlling the protective cover 19 aredepicted. The input system 22 includes the reception unit 60, theactuator controlling unit 68, a state information acquisition unit 74, adriving controlling unit 76, and a motion table 78.

The elements indicated as functional blocks that perform variousprocesses in FIG. 20 can be configured, in hardware, from a circuitblock, a memory, and some other LSIs and are implemented, in software,by a program loaded in the memory and so forth. Accordingly, it isrecognized by those skilled in the art that the functional blocks can beimplemented in various forms only from hardware, only from software, orfrom a combination of hardware and software, and they are not restrictedto any of them.

In the following, an example of a controlling technique for theprotective cover 19 is described. First, as a premise, if the powersupply to the robot 10 is turned on, then the actuator controlling unit68 drives the first motor 52 and the second motor 54 such that the firstarc-shaped arm 32 and the second arc-shaped arm 34 are placed into astate in which they are erected uprightly by 90 degrees with respect tothe pedestal 30. Determining the posture of the housing 20 at this timeas a reference posture, the rotational angle φx of the first motor 52,the rotational angle φy of the second motor 54, and the rotational angleφz of the third motor 56 are set to reference values (for example, 0).In short, in the reference posture, (φx, φy, φz) are set to (0, 0, 0).

FIG. 21(a) to FIG. 21(c) depict positional relationship between ahousing side plate and the protective cover 19.

While the power supply to the robot 10 is turned off, the protectivecover 19 is disposed at the closing position at which it protects thecameras 14. FIG. 21(a) depicts a state in which the protective cover 19covers the front face of the housing 20 thereby to cover the cameras 14.

If the power supply to the robot 10 is turned on, then the drivingcontrolling unit 76 controls rotation of the driving motor 142 to movethe protective cover 19 to its operation initial position. The operationinitial position is a position when the protective cover 19 is rotatedupwardly by 90 degrees from the closing position. FIG. 21(b) depicts astate in which the protective cover 19 is moved to the operation initialposition. Where the protective cover 19 moves to the operation initialposition, the cameras 14 can thereafter pick up an image in the frontdirection. It is to be noted that the operation initial position may beset to a position that is not included in the angle of view of thecameras 14.

After the power supply to the robot 10 is turned on, the protectivecover 19 can move within a range of the operation initial position to anoperation maximum position. However, the protective cover 19 may bemovable within a range of the closing position to the operation maximumposition.

FIG. 21(c) depicts a state in which the protective cover 19 is moved tothe operation maximum position. Although the operation maximum positionmay be set to a rotation limit position of the protective cover 19 atwhich the protective cover 19 and the housing top plate contact witheach other, it may otherwise be set to a position on this side of therotation limit position.

In the following, a technique for controlling rotation of the drivingmotor 142 on the basis of the posture of the housing 20 determined bythe actuator apparatus 12 is described.

The actuator controlling unit 68 controls the rotational angle of thefirst motor 52, the second motor 54, and the third motor 56 inaccordance with sensor information from the HMD 100. The actuatorcontrolling unit 68 provides rotational angle information (φx, φy, φz)indicative of a posture of the housing 20 to the driving controllingunit 76, and the driving controlling unit 76 controls rotation of thedriving motor 142 using the rotational angle information (φx, φy, φz).

In particular, the driving controlling unit 76 determines the rotationalangle φ of the driving motor 142 in accordance with a formula givenbelow. It is to be noted that the rotational angle φ at the operationinitial position is 0.

φ=a×|φx|+b×|φy|+c×|φz|

Here, |φx|, |φy|, and |φz| represent absolute values of the rotationalangles. a, b, and c are weighting coefficients for the rotationalangles, and by suitably setting the weighting coefficients, anoperational characteristic of the protective cover 19 can be determinedaccording to tastes.

Since the driving controlling unit 76 arithmetically operates therotational angle φ on the basis of the rotational angle information (φx,φy, φz) and supplies power to the driving motor 142 such that therotational angle of the driving motor 142 becomes φ, a behavior of theprotective cover 19 interlocking with the posture of the housing 20 canbe implemented.

A different example of the rotation controlling technique based on theposture of the housing 20 is described.

While, in the example described above, the behavior of the protectivecover 19 interlocks with the posture of the housing 20, such control maybe performed that, for example, in the case where the posture of thehousing 20 does not vary for a predetermined period of time, theprotective cover 19 is erected uprightly. When the posture of thehousing 20 does not vary, a situation is assumed in which the user A isconcentrated on some target and is staring at the target. At this time,the driving controlling unit 76 controls the driving motor 142 such thatthe protective cover 19 is erected uprightly. It is to be noted that aperson around the robot 10 preferably knows in advance that uprighterection of the protective cover 19 represents that the user A isconcentrated. By inferring a situation in which the user A isconcentrated from the posture of the housing 20 in this manner, thesituation of the user A can be represented by a motion of the protectivecover 19.

Now, a technique for controlling rotation of the driving motor 142 onthe basis of facial expression information and/or emotion information ofa user detected by the HMD 100 is described.

In the present example, the HMD 100 includes a sensor for detecting amotion of a specific region of the face of the user A. The HMD 100includes a camera in the inside or on the outside of the housing 114 andincludes a facial expression monitoring unit that monitors the facialexpression of the face of a user, more particularly, a motion of an eye,an eyebrow, the nose, a lip or the like.

In the past, although various investigations have been performed inregard to the relationship between an emotion and a facial expression,the following (1) to (5) indicate relationships between an emotion and afacial expression published in previous studies.

-   (1) Happiness: the upper end of the lip rises and is drawn to the    back.-   (2) Surprise: the eyebrows rise and the eyes are wide open.-   (3) Aversion: the upper lip rises and wrinkles are made on the nose.-   (4) Anger: vertical wrinkles are made between the eyebrows, and the    mouth is closed tightly.-   (5) Sorrow: the opposite ends of the lip drop and also the line of    sight rather drops.

A database is made for the relationship between the emotion and thefacial expression in the HMD 100, and the facial expression monitoringunit is configured such that it can refer, if a facial expression of theuser A is specified from a camera image, to the database to read outemotion information corresponding to the facial expression. Since theemotion of a person changes every moment, every time the facialexpression monitoring unit decides that a change in emotion occurs froma change of a specific location included in the camera image, ittransmits emotion information extracted from the database to the robot10.

For example, if the facial expression monitoring unit decides from thecamera image that “the upper end of the lip rises and is drawn to theback,” then it reads out emotion information (“happiness”) associatedwith the facial expression information and transmits the emotioninformation to the robot 10. Thereafter, if the facial expressionmonitoring unit decides from the camera image that “the eyebrows riseand the eyes are wide open,” then it reads out emotion information(“surprise”) associated with the facial expression information andtransmits the emotion information to the robot 10. It is to be notedthat, in the case where the facial expression information detected bythe facial expression monitoring unit is not registered in the database,the facial expression monitoring unit transmits emotion informationindicating that the user has no emotion to the robot 10. It is to benoted that the emotion information to be transmitted may be informationfrom which an emotion can be specified and may be, for example, aninformation identification (ID) of emotion information such as (1) or(2).

The facial expression monitoring unit may transmit, in place of emotioninformation, facial expression information itself, for example, facialexpression information indicating that “the upper end of the lip risesand is drawn to the back” or facial expression information indicatedthat “the eyebrows rise and the eyes are wide open” to the robot 10.Further, the facial expression monitoring unit may transmit both emotioninformation and facial expression information to the robot 10. In thefollowing description, such facial expression and/or emotion informationare collectively referred to as state information.

Referring to FIG. 20, the reception unit 60 receives the stateinformation, and the state information acquisition unit 74 acquires thestate information. The driving controlling unit 76 controls rotation ofthe driving motor 142 on the basis of the state information. At thistime, the driving controlling unit 76 refers to the motion table 78.

The motion table 78 retains facial expression information and/or emotioninformation and motion modes of the protective cover 19 in an associatedrelationship with each other.

FIG. 22 depicts an example of the retained substance of the motiontable. In the motion table 78 depicted in FIG. 22, emotion informationand motion modes of the protective cover 19 are recorded in anassociated relationship with each other. The driving controlling unit 76drives the driving motor 142 on the basis of the emotion informationacquired from the HMD 100 such that the protective cover 19 moves in acorresponding motion mode. For example, in the case where the emotioninformation is “happiness,” the driving controlling unit 76 controlsrotation of the driving motor 142 such that the protective cover 19moves back and forth by 30 degrees forwardly and rearwardly (with anamplitude of 60 degrees) from the state in which the protective cover 19is erected uprightly by 90 degrees from the operation initial position.Consequently, a person around the robot 10 can recognize by viewing themotion of the protective cover 19 that the user A feels happy. Forexample, in the case where the facial expression monitoring unit in theHMD 100 can decide a degree that represents whether the emotion is greator small, the driving controlling unit 76 may set the speed of the backand forth movement in response to the degree.

While, in FIG. 22, the motion table 78 retains emotion information andmotion modes of the protective cover 19 in an associated relationshipwith each other, it may otherwise retain facial expression informationand motion modes of the protective cover 19 in an associatedrelationship with each other.

It is to be noted that, in the example described above, the stateinformation acquisition unit 74 acquires facial expression informationand/or emotion information of a user detected by the HMD 100 the userwears. Although facial expression information and/or emotion informationof a user who wears the HMD 100 may be detected by the HMD 100 in thismanner, it may otherwise be detected by an information processingapparatus provided outside the HMD 100. In other words, although thefacial expression monitoring unit may be provided on the HMD 100, it mayotherwise be provided in an information processing apparatus differentfrom the HMD 100. At this time, the information processing apparatusspecifies a facial expression or an emotion of the user on the basis ofa camera image picked up by a camera that picks up an image of the userand transmits the specified facial expression or emotion to the robot10. In either case, the state information acquisition unit 74 acquiresfacial expression information and/or emotion information of the user whowears the HMD 100, and the driving controlling unit 76 controls rotationof the driving motor 142 on the basis of the facial expressioninformation and/or emotion information of the user.

REFERENCE SIGNS LIST

1 . . . Information processing system, 10 . . . Robot, 12 . . . Actuatorapparatus, 14 a . . . Right camera, 14 b . . . Left camera, 16 a . . .Right microphone, 16 b . . . Left microphone, 20 . . . Housing, 22 . . .Input system, 24 . . . Output system, 30 . . . Pedestal, 32 . . . Firstarc-shaped arm, 32 a . . . First elongated through-hole, 34 . . . Secondarc-shaped arm, 34 a . . . Second elongated through-hole, 36 . . .Housing, 38 . . . Cover, 40 . . . Leg unit, 42 . . . Insertion member,42 a . . . First restriction portion, 42 b . . . Second restrictionportion, 42 c . . . Stem portion, 50 . . . Driving mechanism, 52 . . .First motor, 54 . . . Second motor, 56 . . . Third motor, 60 . . .Reception unit, 62 . . . Sensor information acquisition unit, 64 . . .Motion detection unit, 66 . . . Gaze direction determination unit, 68 .. . Actuator controlling unit, 70 . . . Sound data acquisition unit, 72. . . Sound processing unit, 74 . . . State information acquisitionunit, 76 . . . Driving controlling unit, 78 . . . Motion table, 80 . . .Image processing unit, 82 . . . Sound processing unit, 82 a . . . Phasedifference amplification apparatus, 84 a . . . First amplifier, 84 b . .. Second amplifier, 86 a . . . First adder, 86 b . . . Second adder, 88a . . . Third amplifier, 88 b . . . Fourth amplifier, 90 . . .Transmission unit, 92 . . . Image recording apparatus, 100 . . . HMD,102 . . . Display panel, 104 . . . Earphone, 106 . . . Microphone, 108 .. . Mounting band, 110 . . . Output mechanism unit, 112 . . . Mountingmechanism unit, 114 . . . Housing, 120 . . . Control unit, 122 . . .Storage unit, 124 . . . Posture sensor, 126 . . . Communicationcontrolling unit, 142 . . . Driving motor, 144 . . . Transmissionmember, 200 . . . Processing apparatus, 202 . . . Reception unit, 204 .. . Sensor information acquisition unit, 206 . . . Motion detectionunit, 208 . . . Gaze direction determination unit, 210 . . . Imagedetermination unit, 212 . . . Sound determination unit, 214 . . .Viewing data provision unit, 216 . . . Transmission unit, 218 . . .Recording unit, 220 . . . Image recording unit, 222 . . . Soundrecording unit.

INDUSTRIAL APPLICABILITY

The present invention can be utilized in the field of a robot and soforth.

1. A robot that includes an actuator apparatus and a housing whoseposture can be changed by the actuator apparatus and in which thehousing at least includes a camera, a microphone, and a speaker and theactuator apparatus causes a posture of the housing to interlock with amotion of a head-mounted display apparatus a user wears, the robotcomprising: a movable member supported for motion on the housing; amotor provided in the housing; a transmission member configured totransmit rotation of the motor to the movable member; and a drivingcontrolling unit configured to control rotation of the motor.
 2. Therobot according to claim 1, wherein the housing configures a face of therobot.
 3. The robot according to claim 1, wherein the drivingcontrolling unit controls rotation of the motor on the basis of aposture of the housing.
 4. The robot according to claim 1, wherein thetransmission member is a member that deforms torsionally andelastically.
 5. A housing that includes a camera, a microphone, and aspeaker, the housing comprising: a movable member supported for motionon the housing; a motor; a transmission member configured to transmitrotation of the motor to the movable member; and a driving controllingunit configured to control rotation of the motor.
 6. The housingaccording to claim 5, wherein the transmission member is a member thatdeforms torsionally and elastically.
 7. The housing according to claim6, wherein the transmission member is a coil spring.
 8. The housingaccording to claim 5, wherein the movable member is moved between aclosing position at which the movable member protects the camera and aposition different from the closing position.
 9. The housing accordingto claim 5, wherein the movable member is supported for pivotal motionon the housing.